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Abstract 


The  current  DoD  policy  for  Acquisition  Reform  uses  Commercial-Off-The-Shelf  (COTS) 
products  whenever  possible  to  avoid  development  costs  of  new  systems  acquisition.  The  Next 
Generation  Instrumentation  Bus  (NexGenBus)  project  is  currently  evaluating  commercial 
standards  for  a  high-speed  instrumentation  bus  standard.  Since  commercial  vendors,  institutes, 
and  groups  design  these  standards  for  commercial  applications,  DoD  cannot  blindly  adopt  one  of 
these  standards  for  use  in  military  environments.  The  standard  should  be  tested,  evaluated,  and 
possibly  changed  based  on  the  application.  Using  COTS  products  to  evaluate  a  commercial 
standard  can  be  tricky — if  a  unit  fails,  which  is  at  fault?  From  a  standards  perspective, 
deviations  are  never  desired,  though  they  are  required  in  some  circumstances.  There  is  a  fine 
line  between  adapting  a  commercial  standard  to  make  it  useable  for  a  specific  application  (e.g. 
test  instrumentation)  and  creating  a  non-conforming  variation  to  the  standard. 
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1  introduction 

The  increased  fusion  of  data  from  numerous  sources  (i.e.  analog  measurements,  digital  buses, 
digital  radar  data,  and  digitized  video)  will  overwhelm  the  traditional  approach  of  adding 
instrumentation  busses  whenever  more  bandwidth  is  required.  A  single  standard  instrumentation 
bus  with  data  rates  at  least  10  to  40  times  that  of  current  systems  is  required.  To  comply  with 
Acquisition  Reform  and  with  the  emphasis  on  COTS  hardware,  the  instrumentation  community 
requires  a  commercial  standard  for  the  next  generation  high-speed  instrumentation  bus  standard. 

Today  there  are  no  high-speed  commercial  instrumentation  busses  on  the  market  therefore 
choosing  an  instrumentation  standard  is  not  a  matter  of  choosing  a  commercial  bus  with  the 
widest  installed  user  base,  the  most  capable  system,  or  the  cheapest  components.  Any 
conunercial  bus  chosen  must  be  evaluated  for  use  in  the  test  and  evaluation  environment.  The 
test  and  evaluation  environment  includes  temperature  vibration,  timing,  topologies,  installation, 
and  maintenance  issues.  The  completion  of  this  evaluation  results  in  a  profile  for  use  by  the 
instrumentation  vendors.  In  its  simplest  form,  the  profile  would  say  “See  Standard  xyz”.  More 
likely  the  profile  will  limit  the  number  of  allowed  options,  tighten  certain  parameters,  and  change 
some  of  the  physical  attributes. 

2  Choosing  a  Standard 

The  shift  of  DoD  to  COTS  products  is  necessitating  a  hard  look  at  using  these  products  in  more 
stringent  environments.  Choosing  a  COTS  product  to  use  requires  a  thorough  knowledge  of  the 
application.  The  same  is  true  in  the  standards  arena.  A  standard  is  defined  as  “something 
established  by  authority,  custom,  or  general  consent  as  a  model  or  example”.  DoD  must  alter  its 
approach  to  standards  from  that  of  authority  to  one  of  consent  in  order  to  allow  the  use  of  new 
COTS  products.  The  number  of  standards  available  requires  a  thorough  search  for  the  best  fit.  It 
would  seem  that  more  available  standards  should  provide  a  better  opportunity  to  find  the  exact 
standard  required.  However,  the  system  designer  must  consider  cost  and  schedule  as  well  as 
performance.  The  best  technical  fit  may  not  provide  the  best  overall  solution.  The  life  cycle  cost 
of  implementation  is  getting  more  attention  for  example,  when  there  is  conununity  support  for  a 
standard  the  overall  costs  decrease. 

The  commercial  market  is  driven  by  profit.  DoD  personnel  must  grasp  this  to  understand  the 
‘why  and  how’  new  standards  are  developed.  New  standards  are  proposed  to  allow  multiple 
vendors  to  develop  interoperable  equipment.  The  companies  that  provide  the  best  fit  to  a  new 
standard  early  in  the  process  (Le.  they  wrote  the  standard)  stand  to  make  the  most  profit.  The 
result  is  a  proliferation  of  promising,  but  competing  standards.  Conversely,  once  a  company  has 
staked  out  ground  in  a  successful  standard,  they  fight  to  keep  the  standard  evolving  rather  than 
backing  a  new  one.  A  new  or  revised  standard  is  written  to  replace  its  outdated  parent  or  to 
broaden  the  influence  of  its  sibling.  There  are  many  standards  to  choose  from  in  various  states  of 
maturity.  How  should  one  be  chosen? 

The  task  of  choosing  a  standard  is  not  as  easy  as  it  first  appears.  There  are  many  aspects  that 
need  to  be  considered.  The  relative  importance  of  each  will  be  dependent  upon  the  application 
and  the  reviewer.  Most  likely  there  will  be  one  or  more  fundamental  performance 
characteristics.  Using  these  performance  characteristics,  it  is  important  to  narrow  the  field  down 
as  quickly  as  possible  by  applying  a  broad  filter.  Once  the  number  of  candidates  is  down  to 
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some  manageable  number,  some  intangible  characteristics  need  to  be  evaluated  along  with  the 
remaining  performance  issues.  The  performance  issues  are  specific  to  each  application  and  will 
not  be  discussed.  However,  there  are  a  few  criteria  that  should  be  considered  along  with 
performance  characteristics.  These  criteria  will  help  balance  a  purely  technical  decision  with 
some  thought  to  commercial  viability. 

In  broad  terms,  the  criteria  used  along  with  performance  characteristics  can  be  stated  as  follows: 
standards  viability,  commercial  acceptance,  and  scope.  Once  selected  for  an  application, 
standards  must  stay  in  place  for  years  to  be  effective.  As  such,  it  is  important  to  understand  the 
maturity  level  of  the  standard.  A  standard  that  was  written  last  year  may  contain  the  latest  bells 
and  whistles,  but  may  lack  the  maturity  needed  to  resolve  each  vendors  interpretation  of  the 
standard,  thus  allowing  devices  that  followed  the  standard  to  still  not  be  fully  interoperable. 
Conversely,  a  standard  that’s  been  in  use  for  many  years  may  have  been  modified  superbly,  but 
because  of  technology  advances  may  be  nearing  the  end  of  its  useful  life.  The  activity  level  of 
particular  standards  working  groups  may  also  temper  the  maturity  aspect  of  a  standard.  No 
activity  in  a  working  group  could  indicate  a  lack  of  interest  in  evolving  the  standard.  If  the 
application  is  somewhat  unique,  some  minor  adjustments  to  the  standard  may  be  required.  The 
safest  method  to  evaluate  a  standard  is  to  attend  the  working  group  meetings  sponsored  by  the 
standard. 

Another  aspect  of  standards  evaluation  is  the  return  on  investment.  One  of  the  best  ways  to 
evaluate  this  is  to  research  the  market  place  to  see  what  is  being  offered  for  sale.  Are  multiple 
vendors  or  one  vendor  offering  the  products?  How  do  prices  for  similar  units  using  competing 
standards  compare?  What  articles  are  being  written  about  these  standards  in  the  trade  journals? 
Performing  this  analysis  once  will  highlight  which  standards  are  in  use  today.  Analyzing  this 
analysis  over  several  months  will  give  trend  data  to  assess  which  standards  dominate  the  market 
today  and  which  hold  the  most  promise  for  tomorrow. 

After  an  initial  list  of  standards  has  been  reduced  to  a  manageable  few,  scope  will  still  be  an 
important  consideration.  Several  of  the  resulting  standards  may  perform  adequately. 
Understanding  the  scope  of  the  application  and  the  nuances  of  the  various  standards  may  tip  the 
balance  toward  one  standard  or  another. 

3  Testing  a  Commercial  Standard 

The  applicability  of  a  standard  to  an  application  requires  an  understanding  of  the  critical 
characteristics  of  both  the  standard  and  the  application.  Testing  the  standard  is  a  matter  of 
ensuring  the  standard  works  in  the  new  applications  environment  and  the  new  application  can  be 
handled  by  the  standard.  The  critical  parameters  are  not  necessarily  the  same  for  both,  however, 
a  percentage  probably  overlap.  It  is  this  complete  set  of  critical  parameters  that  must  be 
addressed  in  testing.  There  are  three  questions  that  need  to  be  asked  -  what  to  test,  how  to  test, 
and  what  happens  when  it  doesn’t  work. 

Leading  commercial  standards  will  generally  have  some  commercial  products  available  that  can 
be  purchased  to  use  as  COTS  Units  Under  Test  (CUUT).  COTS  units  used  to  verify  a 
commercial  standard  begs  an  interesting  question.  What  is  being  tested?  Most  commercial 
standards  approved  by  one  of  the  national  standards  organizations  have  had  significant  amounts 
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of  testing.  Therefore,  testing  to  ensure  the  standards  document  is  correct  doesn’t  make  a  lot  of 
sense.  Unless  the  CUUT  is  ruggedized  for  use  in  the  military  environment,  finding  its 
deficiencies  doesn’t  provide  any  answers  as  to  the  standards  applicability  in  this  new 
environment. 

Another  question  that  needs  to  be  answered  is  “How  to  test’’.  When  using  a  CUUT,  development 
test  points  are  unavailable.  Testing  individual  components  is  not  always  an  option  and  testing 
the  unit  as  a  system  doesn’t  provide  any  insight  as  to  which  component  failed.  Without  the 
knowledge  of  what  failed  and  why,  it’s  not  clear  whether  the  standard  wasn’t  written  correctly  or 
the  CUUT  was  at  fault.  It  also  isn’t  clear  whether  either  could  be  modified  to  work  in  this 
environment. 

Finally,  there  are  three  options  if  the  tests  should  fail.  For  minor  problems,  the  requirement 
should  be  reexamined.  Can  the  job  still  be  accomplished  if  the  requirement  is  relaxed?  If  it  can, 
this  is  the  preferable  approach.  If  the  requirement  is  hard,  the  next  thing  to  consider  is  changing 
the  standard.  CAUTION,  this  shouldn’t  be  done  lightly.  Indiscriminately  modifying  the 
standard  may  sacrifice  everything  a  commercial  standard  has  to  offer.  However,  sometimes 
there  are  good  reasons  to  change  a  standard.  The  third  option  is  to  try  a  different  standard. 

The  answer  to  testing  commercial  standards  using  CUUTs  is  a  mix  of  analysis,  lab  testing,  and 
simulation.  Critical  areas  that  cannot  be  readily  tested  should  be  analyzed.  Testing  should  be 
used  where  it  is  required  and  in  support  of  simulation.  Simulation  is  used  to  provide  results 
where  testing  is  too  difficult  or  expensive  and  to  extrapolate  the  results  of  analysis  and  testing  to 
the  application  environment. 


4  The  NexGenBus  Approach 

4.1  Goal 

The  goal  of  the  NexGenBus  Project  is  to  establish  a  commercial  communications  bus  as  a 
standard  for  the  test  instrumentation  system  of  the  future. 

4.2  Understanding  the  Requirements 

Before  evaluating  the  commercial  busses,  the  system  features  of  a  composite  future  data 
acquisition  system  were  identified.  The  composite  data  acquisition  system  included  elements  of 
data  systems  existing  today  along  with  future  elements  envisioned  by  the  NexGenBus  team. 
This  enabled  the  NexGenBus  team  to  understand  the  type  of  data  acquisition  system  a  future 
instrumentation  bus  must  support.  One  of  the  major  features  of  such  a  system  is  the  ability  to 
bridge  to  data  acquisition  units  on  other  busses  allowing  existing  instrumentation  inventories  to 
be  used.  Other  features  included  open  system  architecture,  simultaneous  sampling,  various  data 
inputs  and  outputs,  smart  transducer  support,  environmental  constraints,  and  network  topology. 

The  composite  data  acquisition  system  definition  was  used  as  a  framework  to  define  the 
NexGenBus  requirements.  This  method  enabled  the  team  to  track  the  NexGenBus  requirements 
to  a  speeifie  data  acquisition  element.  For  each  data  acquisition  element  the  required  bus 
eharacteristic  was  determined.  The  bus  requirements  list  included  quantitative  instrumentation 
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measurements  such  as  vibration  and  acoustic  sampling  requirements,  digital  data  (serial  and 
parallel),  video  (weapons  release)  and  audio  requirements.  As  insights  and  information  were 
gained  on  the  bus  characteristics,  the  requirements  list  was  updated. 


4.3  Bus  Standard  Selection 

A  comprehensive  search  was  implemented  for  non-proprietary  communication  busses.  This 
search  entailed  generic  open  searches  on  the  web  using  several  of  the  more  prominent  search 
engines.  This  was  closely  followed  by  thorough  searches  of  standards  organizations  (IEEE, 
ANSI,  etc.).  Trade  journals  and  technical  magazines  provided  timely  information  on  trends  and 
busses  used  in  the  commercial  industry.  The  search  turned  up  more  than  thirty-three  busses. 

Some  of  the  larger  instrumentation  systems  in  use  today  have  data  rates  exceeding  50  Mbps 
through  the  use  of  multiple  busses  and  multiplexers.  The  correlation  of  time  sensitive  data 
across  these  multiple  busses  can  be  tricky  at  best.  To  ease  this  burden,  there  is  a  trend  of 
combining  numerous  sources  (video/data/voice)  onto  one  very  fast  bus.  With  this  in  mind,  the 
minimum  requirement  for  bus  speed  was  considered  to  be  an  order  of  magnitude  greater  than  the 
current  10  Mbps  standard.  Eight  serial  busses  with  rates  greater  than  100  Mbps  were  identified. 
These  eight  busses  were  selected  for  review  and  grading.  The  purpose  of  this  stage  was  to 
quickly  eliminate  those  busses  that  could  not  reasonably  be  used  in  an  instrumentation 
environment.  An  evaluation  sheet  was  generated  based  on  some  of  the  identified  critical 
tequirements.  The  final  grading  on  the  eight  busses  showed  three  clear  leaders.  Fibre  Channel, 
FireWire  and  Gigabit  Ethernet 

The  resultant  three  busses  were  studied  in  more  detail  This  next  phase  focused  on  whether  the 
busses  could  perform  critical  instrumentation  related  tasks  such  as  class  of  service,  latency,  and 
synchronicity.  These  busses  were  judged  on  market  as  well  as  performance  factors.  The  down- 
select  criteria  for  this  third  cut  considered  thirteen  items  within  three  main  areas:  viability, 
conunercial  acceptance,  and  technical  scope.  Fibre  Channel  was  the  winner  again.  This 
evaluation  supports  what  is  being  seen  in  other  DoD  activities.  Of  the  three  busses.  Fibre 
Channel  is  the  only  one  being  used  in  a  military  flight  environment. 

4.4  Testing 

The  Fibre  Channel  standard  is  structured  as  five  functional  levels  identified  as  FC-0  through  PC-4. 
These  levels  are  based  on  the  lowest  four  layers  of  the  seven-layer  Open  System  Interconnect  (OSI) 
model  used  throughout  the  communications  industry.  Testing  Fibre  Channel  within  its  functional 
levels  provides  a  solid  base  for  the  test  program. 

The  goal  of  the  test  program  is  to  identify  weaknesses  in  areas  critical  to  instrumentation.  However, 
because  we  are  testing  COTS  end-items,  we  may  be  unable  to  isolate  the  area  under  test.  Therefore 
we  shall  test  a  function  over  several  layers  and  interpolate  the  results.  The  major  areas  of  the  test 
program  are  lab  testing  and  simulation. 

4.4.1  Lab  testing 

The  requirements  used  in  the  evaluation  portion  of  the  down  select  were  used  as  the  theoretical 
“best”  areas  in  which  to  test  the  Fibre  Channel  layers.  The  lab  set-up  however  required  a 
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mapping  from  the  theoretical  “best”  areas  to  the  actual  layer  in  which  the  tests  were  performed. 
For  example,  the  data  rate  in  Fibre  Channel  is  defined  at  the  FC-0  (physical)  layer.  The  actual 
testing  of  the  data  rate  was  tested  at  the  FC-4  (application)  layer.  Whenever  possible,  ways  to 
work  within  the  standard  to  avoid  these  weaknesses  were  noted  and  allowed  the  test  team  to 
track  the  areas  of  deviation  from  the  standard  for  possible  analysis  at  a  later  date.  The  selected 
tests  were  oriented  towards  operational  requirements.  Development  tests  were  considered  as  a 
secondary  objective  and  oriented  toward  ensuring  the  operational  requirements.  The  testing 
activity  will  be  accomplished  in  two  stages.  The  first  stage  is  oriented  toward  port  testing  and 
the  second  stage  is  oriented  toward  node  testing.  Each  of  these  stages  is  composed  of  multiple 
test  categories.  The  objective  of  the  port  testing  is  to  establish  benchmark  tests  and  to  determine 
if  the  port  frmctions  meet  the  operational  requirements.  The  exit  criterion  for  this  stage  is  based 
on  the  effectiveness  of  the  port  functions  to  meet  the  operational  requirements.  The  objective  of 
the  node  testing  is  to  build  upon  stage  one  by  testing  selective  protocols.  The  protocols  selected 
will  be  the  results  from  the  protocol  simulations.  The  entrance  criterion  for  this  stage  is  based  on 
the  effectiveness  of  the  protocol  simulations  for  the  operational  requirements.  The  exit  criterion 
is  based  on  the  node  functions  meeting  the  operational  requirements. 

4.4.2  Simulation 

There  are  areas  within  a  standard  which  are  better  left  to  simulation.  These  areas  include  new 
protocols,  loading  analysis,  flow  control.  Bit  Error  Rate  (BER)  and  error  correction.  Simulating 
a  protocol  or  introducing  errors  in  a  simulation  may  be  much  more  cost  effective  than  doing 
actual  testing.  ‘Test  where  you  must.  Simulate  where  you  can”  defines  a  philosophy  whereby 
the  requisite  test  points  are  done  in  the  lab  but  the  difficult  or  impossible  test  points  are  done  in  a 
simulation.  This  complements  the  lab  testing  and  saves  the  cost  of  expensive  test  equipment. 
However,  since  a  simulation  is  only  as  good  as  the  parameters  that  are  input,  lab  testing  is 
required  to  provide  verification  of  the  simulation. 

There  are  multiple  protocols  that  can  be  run  on  top  of  Fibre  Channel.  Part  of  the  test  program 
includes  simulating  a  data  acquisition  network  using  various  protocols  on  top  of  Fibre  Channel  in 
order  to  select  the  best  one(s).  The  simulation  results  will  help  to  determine  what  topology  and 
protocols  are  best  suited  for  the  NexGenBus.  The  requirements  that  will  be  modeled  are  Data 
Rate,  Synchronicity,  and  Latency.  Data  will  be  collected  and  compared  for  several  protocols. 
The  effect  of  topology  on  these  criteria  will  also  be  investigated.  We  will  simulate  a  series  of 
file  transfers  and  record  the  message  delay  and  transfer  rates.  This  will  be  done  for  several 
protocols  and  classes  of  service.  The  results  from  the  lab  will  be  compared  to  those  from  the 
simulations  to  verify  the  accuracy  of  the  model.  Any  discrepancies  will  be  analyzed  and  the 
model  will  be  adjusted  as  necessary  to  best  simulate  the  actual  operation  of  Fibre  Channel.  Once 
the  model  has  been  verified  it  will  become  our  baseline  model.  Due  to  the  nature  of  the  baseline 
model,  synchronicity  will  not  be  measured  in  the  initial  simulations. 

Once  the  baseline  model  is  established,  we  will  proceed  to  experiment  with  topologies.  These 
topologies  will  include  Arbitrated  Loop,  Meshed  Fabric,  and  other  combinations.  Using  the 
modeling  approach  provides  the  flexibility  to  perform  “what-if’  type  scenarios  with  the  bus 
architecture,  such  as,  what  if  a  link  becomes  over  utilized  or  goes  down  altogether. 
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5  Summary 

There  is  leverage  to  be  gained  from  using  COTS  products  and  standards  in  DoD  applications. 
New  technology  can  be  incorporated  much  more  quickly  and  cheaply.  However,  all  of  the 
advantages  of  using  a  commercial  standard  in  a  program  can  be  quickly  reduced  to  zero  if  used 
improperly.  The  requirements  of  the  application  and  the  limitations  of  the  standard  must  be 
understood  and  taken  into  account. 
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